[SPARK-51473][ML][CONNECT] ML transformed dataframe keep a reference to the model - #50199
[SPARK-51473][ML][CONNECT] ML transformed dataframe keep a reference to the model#50199zhengruifeng wants to merge 7 commits into
Conversation
dd3b334 to
2a607be
Compare
zhengruifeng
changed the title
2b78d14 to
7058de2
Compare
There was a problem hiding this comment.
what's the previous ConnectDataFrame for ?
There was a problem hiding this comment.
ConnectDataFrame is to build the connect DF, it is still used at the end of this method
There was a problem hiding this comment.
For model summary, it might also return Dataframe, shall we make similar change ?
7058de2 to
af525d4
Compare
There was a problem hiding this comment.
case 1: summary.predictions
the DF predictions hold a reference to summary (which link to the model)
case 2: GMMModel.gaussiansDF
the DF gaussiansDF hold a reference to GMMModel
There was a problem hiding this comment.
training summary hold a reference to the model
There was a problem hiding this comment.
the testing summary model.evaluate(df) hold a reference to the model
af525d4 to
05f37bc
Compare
|
ping @WeichenXu123 |
|
merged to master |
2 participants
What changes were proposed in this pull request?
add the model link in the transformed dataframe
Why are the changes needed?
#49948 disabled the model GC for
fit_transform, this PR add the model link in the transformed dataframe, so that the model will be GCed together with the transformed dataframeDoes this PR introduce any user-facing change?
no
How was this patch tested?
existing test should cover this change
Was this patch authored or co-authored using generative AI tooling?
no